Skip to content

test(e2e): improve integration test reliability#8422

Merged
jacekradko merged 2 commits into
mainfrom
jacek/fix-e2e-compromised-password
May 1, 2026
Merged

test(e2e): improve integration test reliability#8422
jacekradko merged 2 commits into
mainfrom
jacek/fix-e2e-compromised-password

Conversation

@jacekradko
Copy link
Copy Markdown
Member

@jacekradko jacekradko commented Apr 30, 2026

Summary

Three small E2E reliability fixes uncovered while investigating consistent CI failures across multiple suites.

1. Strong random fake-user passwords

createFakeUser built passwords as ${email}${randomHash} (e.g. 76557538ea+clerk_test@clerkcookie.com76557538ea). FAPI's compromised-password check rejects these with form_password_compromised (HTTP 422), which broke any sign-in flow that wasn't intentionally testing the compromised-password path. Replaced with a fakerPassword() helper that emits a high-entropy base64url string with a fixed Aa1! prefix to satisfy default Clerk complexity rules (upper, lower, digit, symbol).

2. Defensive afterAll cleanup

When a test suite's beforeAll timed out or threw (BAPI hiccup, dev server slow to come up, etc.), afterAll would crash with a TypeError on the un-assigned fakeUser / fakeOrganization / network / fakeOAuth / fakeAPIKey, masking the real failure. Optional-chain the cleanup calls in:

  • integration/tests/components.test.ts
  • integration/testUtils/machineAuthHelpers.ts (apiKey, m2m, oauth describes)

3. Drop timer-dependent token-cache assertions

Two MemoryTokenCache cross-tab tests assumed JWT TTL = 60s and waited 50s of wall-clock time for the proactive-refresh setTimeout to fire. The dev test instance now issues 300s tokens, so the refresh fires at ~283s and the tests never reached it.

The BroadcastChannel-based deduplication these tests were checking is already covered elsewhere; the proactive-refresh timer scheduling itself is best validated by unit tests that mock setTimeout, not by real-time integration tests.

  • Delete integration/tests/session-token-cache/single-session.test.ts "multi-tab scheduled refreshes are deduped to a single request". The same broadcast-dedup path is exercised by the existing line-49 test ("multi-tab token sharing works when clearing the cache") via getToken({ skipCache: true }).
  • Rewrite integration/tests/session-token-cache/multi-session.test.ts:249 (now "cross-session token refreshes do not deduplicate") to manually trigger getToken({ skipCache: true }) on each tab. Different sessions have different tokenIds, so BroadcastChannel doesn't dedupe — each tab is expected to make its own request. Unique coverage the single-session test doesn't provide.

Scoped to integration/ only and doesn't affect any published package — empty changeset.

Test plan

  • generic components-component-smoke — real beforeAll failure surfaces instead of TypeError: ... 'delete'
  • machine tests-with-invalid-M2M-tokens / local-verification — real ClerkAPIResponseError surfaces instead of TypeError: ... 'cleanup'
  • generic multi-tab token sharing works when clearing the cache — still green (covers the broadcast-dedup path the deleted timer test was duplicating)
  • nextjs cross-session token refreshes do not deduplicate — green; runs in seconds and asserts the cross-session non-dedup case
  • No regression in nextjs session-tasks-sign-in-reset-password (test still calls setPasswordCompromised on purpose)

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 30, 2026

🦋 Changeset detected

Latest commit: 6cf9d6c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 30, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
clerk-js-sandbox Ready Ready Preview, Comment May 1, 2026 0:11am

Request Review

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented Apr 30, 2026

Open in StackBlitz

@clerk/astro

npm i https://pkg.pr.new/@clerk/astro@8422

@clerk/backend

npm i https://pkg.pr.new/@clerk/backend@8422

@clerk/chrome-extension

npm i https://pkg.pr.new/@clerk/chrome-extension@8422

@clerk/clerk-js

npm i https://pkg.pr.new/@clerk/clerk-js@8422

@clerk/dev-cli

npm i https://pkg.pr.new/@clerk/dev-cli@8422

@clerk/expo

npm i https://pkg.pr.new/@clerk/expo@8422

@clerk/expo-passkeys

npm i https://pkg.pr.new/@clerk/expo-passkeys@8422

@clerk/express

npm i https://pkg.pr.new/@clerk/express@8422

@clerk/fastify

npm i https://pkg.pr.new/@clerk/fastify@8422

@clerk/hono

npm i https://pkg.pr.new/@clerk/hono@8422

@clerk/localizations

npm i https://pkg.pr.new/@clerk/localizations@8422

@clerk/nextjs

npm i https://pkg.pr.new/@clerk/nextjs@8422

@clerk/nuxt

npm i https://pkg.pr.new/@clerk/nuxt@8422

@clerk/react

npm i https://pkg.pr.new/@clerk/react@8422

@clerk/react-router

npm i https://pkg.pr.new/@clerk/react-router@8422

@clerk/shared

npm i https://pkg.pr.new/@clerk/shared@8422

@clerk/tanstack-react-start

npm i https://pkg.pr.new/@clerk/tanstack-react-start@8422

@clerk/testing

npm i https://pkg.pr.new/@clerk/testing@8422

@clerk/ui

npm i https://pkg.pr.new/@clerk/ui@8422

@clerk/upgrade

npm i https://pkg.pr.new/@clerk/upgrade@8422

@clerk/vue

npm i https://pkg.pr.new/@clerk/vue@8422

commit: 6cf9d6c

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 30, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

A fix for e2e test reliability is introduced by adding a new fakerPassword helper function that generates strong test passwords using cryptographic randomness and base64 encoding, ensuring compliance with password complexity requirements. The createFakeUser function is updated to use this helper instead of simple email concatenation. Additionally, test teardown logic across multiple test files is made more defensive by applying optional chaining to safely handle cases where test fixtures may be uninitialized or missing.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The PR description comprehensively relates to the changeset, detailing the three reliability fixes with clear explanations of the problems and solutions.
Title check ✅ Passed The title 'test(e2e): improve integration test reliability' is partially related to the changeset. While it accurately describes one aspect of the changes (improving test reliability through defensive cleanup), it does not highlight the main change, which is fixing e2e failures caused by a compromised-password check by introducing a stronger fake password generator.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@jacekradko jacekradko changed the title test(integration): use strong random passwords and harden afterAll test(e2e): use strong random passwords and harden afterAll Apr 30, 2026
@jacekradko jacekradko changed the title test(e2e): use strong random passwords and harden afterAll test(e2e): addressing new e2e failures Apr 30, 2026
@jacekradko jacekradko changed the title test(e2e): addressing new e2e failures test(e2e): use strong fake-user passwords and harden afterAll cleanup Apr 30, 2026
@jacekradko jacekradko changed the title test(e2e): use strong fake-user passwords and harden afterAll cleanup test(e2e): fake-user passwords, defensive afterAll, and virtual clock for token-cache tests Apr 30, 2026
@jacekradko jacekradko changed the title test(e2e): fake-user passwords, defensive afterAll, and virtual clock for token-cache tests test(e2e): fake-user passwords, defensive afterAll, and trim timer-dependent token-cache tests Apr 30, 2026
…ent token-cache tests

Three small E2E reliability fixes uncovered while investigating
consistent CI failures across multiple suites.

1. Strong random fake-user passwords. createFakeUser built passwords
   as `${email}${randomHash}`, which FAPI's compromised-password check
   rejected with `form_password_compromised` (HTTP 422). Replaced with
   a `fakerPassword()` helper that emits a high-entropy base64url
   string with a fixed `Aa1!` prefix to satisfy default Clerk password
   complexity rules.

2. Defensive afterAll cleanup. When a suite's beforeAll timed out or
   threw, afterAll crashed with a TypeError on un-assigned fakeUser /
   fakeOrganization / network / fakeOAuth / fakeAPIKey, masking the
   real failure. Optional-chained the cleanup calls in:
   - integration/tests/components.test.ts
   - integration/testUtils/machineAuthHelpers.ts (apiKey, m2m, oauth)

3. Drop timer-dependent token-cache assertions. Two MemoryTokenCache
   cross-tab tests assumed JWT TTL = 60s and waited 50s of wall-clock
   time for the proactive-refresh setTimeout to fire. The dev test
   instance now issues 300s tokens, so the timer fires at ~283s and
   the tests never reached it.
   - Deleted `multi-tab scheduled refreshes are deduped to a single
     request` from single-session.test.ts. The same broadcast-dedup
     path is exercised by the existing line-49 test via
     `getToken({ skipCache: true })`.
   - Rewrote multi-session.test.ts:249 (now `cross-session token
     refreshes do not deduplicate`) to manually trigger
     `getToken({ skipCache: true })` on each tab. Different sessions
     have different tokenIds, so BroadcastChannel does NOT deduplicate
     across them — unique coverage the single-session test doesn't
     provide.

   Proactive-refresh timer scheduling math is best validated by unit
   tests that mock setTimeout, not by real-time integration tests
   against a shared dev instance.

Scoped to integration/ only, no published package affected — empty
changeset.
@jacekradko jacekradko force-pushed the jacek/fix-e2e-compromised-password branch from 6eebdd2 to ef5dd73 Compare April 30, 2026 21:29
@jacekradko jacekradko changed the title test(e2e): fake-user passwords, defensive afterAll, and trim timer-dependent token-cache tests test(e2e): improve integration test reliability Apr 30, 2026
@jacekradko jacekradko merged commit 1f804dc into main May 1, 2026
132 of 136 checks passed
@jacekradko jacekradko deleted the jacek/fix-e2e-compromised-password branch May 1, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants